Digital light projectors, also commonly known as digital light processors, typically include one or more LED or semiconductor laser light sources, a spatial light modulator such as a digital micromirror device (DMD), driver electronics to convert pre-loaded images and streaming image and video input to timed micromirror actuation, and optics to relay the light source to the spatial light modulator and to then project an image onto an external target. To project an image comprising multiple overlapping colors and intensity levels, for example 24-bit RGB images, the DMD rapidly projects a series of 24 bit plane images. These bit plane images, each called an illumination pattern, can be set to have adjustable pattern periods and illumination exposure times to produce a desired image or video on a target. For example, to obtain 8-bit intensity shading, the projector's DMD is typically configured to display 8 bit plane illumination patterns. Each illumination exposure time and pattern period is typically set to be a factor of two greater than the previous pattern, so that the combined 8 illumination pattern intensity at the target can be set to 256 different levels for each micromirror array element in the DMD. The projector illumination source intensity is usually independently set with its drive current or modulation frequency to further control the illumination level.
Many commercially available projectors are configured to run in a series of fixed operating modes (for example, 24-bit 60 Hz RGB) since they are designed to project images and videos using known image and video formatting standards. Some commercially available projectors can be configured to run in a so-called structured illumination mode, whereby users are given control over the timing and number of illumination patterns through software. In this mode, a standard 24-bit 60 Hz RGB video or pre-loaded images are used as input that can be converted by the driver and displayed as a series of bit planes that are separated in time. An illumination pattern can consist of multiple bit planes, each projected using one or more illumination sources. The illumination patterns can be projected at higher speeds than 60 Hz, depending on the number of bit planes displayed per pattern. Since digital light projectors that incorporate a DMD produce images and videos on a target as a sequence of bit planes, those skilled in the art recognize that the structured illumination mode is simply a more flexible operating mode that can be made available to users with the use of driver electronics that control the illumination source timing and individual bit plane pattern timing.
When switching from one bit plane to the next, there is a certain ‘off-time’ or blanking time, during which time there is no light output from the projector. This blanking time can be inherent in the design of the projector and required for reasons such as resetting the position of the micromirrors in a DMD or for memory read or access functions when loading new DMD pattern data. The blanking time and duty cycle of the illumination in a projected pattern can also be set through software with a pattern period time that is greater than the pattern exposure time.
As an example, the Texas Instruments' DLP LightCrafter product can be configured in the aforementioned structured illumination mode to display a series of 1-bit plane monochrome patterns, either from a 60 Hz 24-bit RGB video input from a standard video display driver, such as used on personal computers, or from up to 96 1-bit per pixel bitmap patterns that have been pre-loaded into its flash memory. When projecting 1-bit plane patterns using the 60 Hz real-time video input, the maximum achievable pattern rate in this configuration is 24*60 Hz=1.44 kHz. When the patterns are preloaded, the projector can operate even faster, obtaining a maximum pattern rate of 4 kHz with a pattern exposure and pattern period time of 250 μsec when fewer than 24 patterns are preloaded. The maximum pattern rate decreases to 2.597 kHz when 24-96 patterns are preloaded due to an additional timing overhead of approximately 135 μsec. Pattern sequences can be customized with multiple output colors and bit depths, and can be configured to run in a loop (i.e. after the last pattern in the sequence, the sequence restarts and the first pattern is displayed).
Other currently available Texas Instruments' DLP and DMD products have similar functionality, with various speed and output specifications. The DLP LightCrafter timing overhead previously mentioned has been reduced in newer versions of Texas Instruments' DLP products so that there is very little inherent time delay between projecting multiple patterns, though the duty cycle can be adjusted with a pattern exposure time that is less than the pattern period. On these projectors, the start of each pattern can be controlled using an external or internal trigger, a strobe output for each pattern or a pattern sequence is typically available, and both the input and output trigger delay, pattern exposure time and pattern period can be set through software. The projectors are comprised of one or more LED or semiconductor laser illumination sources, a DMD, driver electronics, and optics to relay the light from the illumination source(s) to the DMD, and from the DMD to one or more lenses that allow the projection of an image or video onto a target that is external to the projector. Those skilled in the art will recognize that a projector can be custom-built using the above components and set to project a sequence of bit plane illumination patterns.
The rolling shutter method of detection in standard CMOS two dimensional pixel array sensors progressively exposes pixel rows to light during a frame exposure. Unlike the global shutter method of detection, in which all pixels are exposed to light simultaneously, rolling shutter detectors start the exposure of each pixel row at a slightly different time. A commonly stated downside of this method of detection is that it can result in motion artifacts when the target is moving throughout a frame exposure. Some CMOS sensors attempt to overcome this effect by using a global reset, which starts the exposure of all rows simultaneously, but has the undesirable result of having a non-constant exposure time for each row. CMOS rolling shutter sensors typically use a constant pixel clock signal to ensure constant pixel exposure. It is standard for a CMOS sensor to use a line, or horizontal clock, signal to properly acquire and transmit data to the sensor's controller (e.g. a computer). The line signal is usually derived from the sensor clock signal and is at a fixed and constant frequency during a frame exposure.
For the purposes of consistency with existing nomenclature on commercial sensor datasheets and for convenience, the rolling shutter is considered to expose pixels in a row-by-row fashion across the sensor's region of interest during a frame exposure. Rolling shutters on CMOS sensors are typically linear, i.e. they extend across one dimension of the two dimensional sensor. Rolling shutter detection can also be done on a column-by-column basis, or from the center of the 2-dimensional sensor out to the sides, as done on several commercially available scientific CMOS cameras, provided the sensor architecture and drivers support such operation.
The rolling shutter is not a physical (mechanical) shutter, but has been referred in prior art as an electronic shutter or as a spatial filter since only a subset of the total rows in the region of interest is exposed to light at any instant in time. The rolling shutter width is considered to be the number of pixel rows that are exposed at a given instant in time, and is calculated by multiplying the row exposure time by the horizontal clock frequency. Those skilled in the art will recognize that only the light that is incident an active row during the rolling shutter exposure will be detected. Unlike a global shutter sensor, light that is incident on a row during a frame exposure that is not within the instantaneous rolling shutter width will not be detected.
A projector can be configured in its structured illumination mode to rapidly project a sequence of linear illumination patterns onto a target. When the target is imaged by a two-dimensional CMOS sensor that uses a rolling shutter method of detection, the light return from the target can be spatially filtered when temporal and spatial synchronization exists between the rolling shutter exposure and the illumination line patterns. This effect has been documented in prior art for confocal ophthalmic imaging and microscopy. The real-time electronic adjustment of the spatial filtering function, enabled by the CMOS rolling shutter, has been used to effectively remove unwanted scattered light, perform dark-field imaging, polarization sensitive, fluorescence, ophthalmic, multi-wavelength, and interferometric imaging.
When performing confocal imaging with a CMOS rolling shutter and projector, a narrow rolling shutter width is usually desired in order to maximize the benefit of spatial filtering, which can remove unwanted scattered light and other artifacts from the resulting images. Usually, a fixed number of linear illumination patterns (N) is synchronized to the frame exposure of the CMOS sensor. The sensor is triggered every N patterns or, alternatively, the projector is triggered to display N patterns every sensor frame using the horizontal clock signal so that temporal and spatial synchronization between the illumination line and rolling shutter is maintained from frame to frame. A narrow shutter width is typically obtained with a short exposure time so that the frame rate can be maximized. A narrow shutter width can also be obtained with a lower horizontal clock frequency. In this case, the pattern exposure time can be increased while maintaining temporal synchronization, allowing more photons to be detected at a slower frame rate.
The temporal synchronization between the illumination patterns and rolling shutter can be adjusted using the input or output (strobe) trigger delay on either the projector or the sensor. The spatial synchronization can be adjusted by changing the illumination pattern geometry or starting position, as well as be adjusting the starting row of the imaging region of interest on the sensor. When performing dark field imaging, temporal adjustments to the synchronization are typically preferred since they do not cause a spatial shift in image features. Also, on some projectors and CMOS sensors, adjustment to the illumination pattern geometry or the imaging region of interest require an additional timing delay, which can result in a lost image frame.
Although the trigger delay can typically be adjusted on a frame-to-frame basis without causing a loss in the sensor frame rate, an alternative approach to adjusting the temporal synchronization is given in U.S. Pat. No. 8,237,835. In this approach, the sensor and digital light projector are run at slightly different frame rates in order to collect a series of repeating images with known timing offsets. The stated application for this mode of operation is to collect multiply scattered light image data to enhance image contrast or determine scattering properties of the target.
Triangulation-based depth profiling can be performed using the CMOS rolling shutter and projector operated in structured light mode by varying the temporal or spatial offset between the start of the sensor frame relative to the start of the projected patterns. By adjusting the temporal or spatial synchronization through software or hardware, referred to herein as a synchronization offset, the imaging system can be arranged to preferentially detect light return from a specific angle from the target. By quickly sweeping through a range of synchronization offsets (e.g. by changing, prior to the start of each frame, the pixel ROI starting row on the sensor or the trigger delay on the either the projector or sensor) and recording the offset that yielded the maximum pixel intensity, a depth profile of a sample can be quickly obtained with minimal post-processing. A priori calibration with known depth features can be performed to establish the relationship between the synchronization offset and physical depth changes. A key advantage of this system is that the angle of light return is precisely measured and can be rapidly adjusted electronically or through software in small (e.g. pixel or microsecond) increments. Unlike an on-axis confocal imaging system, a triangulation-based system uses the rolling shutter to spatially filter the angle of light return. The width of the illumination pattern lines and rolling shutter width determines the angular resolution of the light return, which is related to the achievable depth resolution. The range of synchronization offsets, combined with the depth of focus at the target, determines the range of resolvable angles of light return from the target, and is related to the measurable depth range. These relationships are expected and consistent with triangulation-based depth profiling theory. One embodiment of a triangulation-based depth profiling microscope using the above approach has been presented by spatially separating the illumination and detection light in a Fourier plane prior to the microscope objective.
When using the rolling shutter for confocal imaging or for triangulation-based depth profiling, a narrow shutter width is usually desired in order to maximize the benefit of spatial filtering, which removes unwanted scattered light and other artifacts from the resulting images. When a sufficiently narrow rolling shutter is used, a periodic series of line artifacts typically becomes visible in the resulting image, despite having good temporal and spatial synchronization across the image frame. These line artifacts are caused by the finite time required by the projector, either from hardware limitations or from the pattern period set by the user, to switch from one pattern to the next. During this pattern switching time period, also called the blanking interval, light is not incident on the sample. However, since the rolling shutter continues to expose pixel rows to light during the frame exposure without interruption, the detected light intensity on the rows during blanking periods will decrease. Prior art has documented this problem by showing the detected row-by-row intensity of a white card. This intensity profile shows a periodic series of lower intensity regions (darker rows), consistent with the blanking period of the projected patterns.
The darker rows in the image can be reduced or overcome by widening the rolling shutter width. If sufficiently wide, the light from adjacent illumination patterns will be collected during exposure, thereby reducing the decrease in intensity during blanking intervals. However, a wider shutter width allows more scattered light to reach the detector, which degrades optical sectioning performance in confocal imaging applications, and depth sectioning performance in triangulation-based depth profiling applications. The decrease in intensity can be modeled as a convolution between the illumination line patterns and the rolling shutter. A method that removes the blanking line artifacts from images while preserving a narrow shutter width for confocal imaging and triangulation-based depth mapping would be appreciated.
When the shutter width is narrow (i.e. smaller than the line pattern width) and the line patterns are projected with a duty cycle of less than 100%, the dark rows will be strongly visible in the images. Since the dark rows repeat after every pattern exposure, the overall image will appear to have a series of spatial fringes, with a frequency related to the number of line patterns that are used to spatially fill the image. These fringes are the result of a spatial-temporal relationship (the convolution) between the illumination lines and the rolling shutter. Unlike prior art that discloses periodic fringe patterns used in structured illumination microscopy and structured light imaging related devices and approaches, the time-averaged illumination is continuous across the field of view since the projected line patterns are adjacent, with no spatial gaps. To emphasize this point, if a global shutter CCD camera were to record a frame with the same exposure time, no fringes would appear in the image, since the fringes are due to the temporal relationship between the projected pattern and rolling shutter.
A common post-processing step required in phase measuring profilometry is that of phase unwrapping. Phase wrapping occurs when the spatial fringes that are projected onto a target appear phase-shifted by more than the fringe half-period at the sensor. To disambiguate the fringe phases for depth mapping, additional computationally post-processing steps are required. A method for phase measuring profilometry that avoids the phase wrapping ambiguity would be appreciated.
Prior art has described several phase measuring profilometry devices and methods that employ the use of a digital light projector to project single or multiple bit plane fringe patterns onto the target. In these systems, slight defocusing can be used to change the spatial intensity profile at the target from a square-wave to a sinusoid. The use of sinusoidal fringe patterns is generally preferred in these systems due to the less computationally complex post-processing steps required to determine the depth of the target features. Square-wave fringe patterns will contain higher order harmonics that are typically not completely removed from the image after processing. In a triangulation-based imaging system that uses projected line patterns that are synchronized to the rolling shutter exposure of a CMOS camera, the rolling shutter acts as a spatial filter to remove light return from different target depths. In this manner, fringes that would normally be phase-shifted due to scattering at different depths are attenuated when using a narrow shutter width, and can be selected with an adjustment to the spatial or temporal synchronization. A method that allows detected fringe patterns to be filtered to remove any residual non-linearity and high order harmonics to obtain a more optimal sinusoidal fringe pattern for use with well-known structured illumination imaging and structured light microscopy post-processing techniques would be appreciated.
The present invention is intended to improve upon and resolve some of these known deficiencies within the relevant art.